Method, system and computer program to optimize deterministic event record and replay

ABSTRACT

A method, system and computer-usable medium for managing task events during the scheduling period of a task executing on one of the CPUs of a multi-processor computer. Only events of specific portions of scheduling period are logged, wherein a first shared resource access has been granted for the task, this portion of scheduling period gathering all the non-deterministic events which cannot be replayed by simple task re-execution. Other independent non-deterministic event records are still logged as usual when they occur out of the portion of scheduling period for which a record has been created. This limits the number of logged events during recording session of an application and the frequency of events to transmit from the production machine to the replay machine.

CROSS-REFERENCE TO FOREIGN APPLICATION

This patent application claims the benefit of the priority date of aprior foreign application filed under 35 U.S.C. § 119, namelycounterpart European Patent Application No. EP07301604.0, entitled “AMethod, System and Computer Program to Optimize Deterministic EvenRecord and Play” which was filed on Nov. 30, 2007 and is incorporatedherein by reference in its entirety.

FIELD OF THE INVENTION

The present invention generally relates to performing record and replayof applications; more particularly the invention improves performancefor record and replay of deterministic events.

BACKGROUND OF THE INVENTION

With record and replay of applications, the goal is to allow thesimultaneous identical execution of an application, for instance, ondifferent machines. This implies that not only the execution has to bereproduced identically, but it must occur at nearly the same time on adifferent host, despite the constraint of being remote (network latency,bandwidth) and with a minimal performance degradation.

On the other hand, the operating systems running on multi-processormachines able to operate in parallel must be adapted in order to allowrecord and replay of an application which is executing non deterministicevents. Between those events, the application execution depends onlyfrom its initial state and program instructions and is, therefore,deterministic. In the case of parallel architecture, such as amulti-processor computer or a network comprising a number of computersrunning in parallel, the use of shared resources accessible by aplurality of tasks adds a cause of non-determinism: the ordering ofaccess to a shared resource by concurrent tasks.

In the simple case where a particular instruction or system call returnsa non-predictable result, it is sufficient to instrument this operationin order to record its result during the original execution and atreplay, to simulate it and to force its result from the recorded value.A set of instructions and system calls which are deterministic onprivate unshared memory become totally non-deterministic when operatingon shared memory because of the uncertainty of the initial state causedby the concurrent use of memory by other tasks, as described above.Rather than instrumenting each and every program instruction, the sameapplicant has proposed a method to ensure the exclusive access to theshared memory during a scheduling period by a single task, thusrestoring the deterministic property of an instruction block, asdescribed in the international patent application ‘Method for optimizingthe logging and replay of multi-task applications in a mono-processor ormulti-processor computer system’ published under the number WO2006/077260. As described in this patent application, during therecording session, one fifo queue per CPU is used for recording eachtask schedule period information and one fifo queue per shared resourceis used for recording each exclusive access to that shared resourceduring task execution. During the replaying session, the logging data offifo queues transmitted to the replay machine are serialized toconstitute the replay scheduling. The events are replayed according tothe replay scheduling on each record from a CPU fifo generating a stopof the corresponding task execution.

A record of a task scheduling period in one CPU fifo contains theinformation on the event having caused task interruption: the event canbe a system call interrupt, a scheduler interrupt or a shared resourceaccess interrupt. At replay, if the event from a CPU fifo is a schedulerinterrupt (called UIC because it uses user instruction count), then aninterrupt is programmed to force the task to stop at the correctinstruction count before resuming the task. The interrupt will be eithertriggered by a performance monitoring counter register overflow (the PMCcounting user instructions) or a software breakpoint. After the taskresumes and suspends again, the task state is matched against theexpected stop condition.

Three possible results can occur from the match. The first possibleresult involves an unexpected scheduler, or breakpoint interrupt, beforethe next stop condition: the task needs simply to be resumed. The secondpossible result involves unexpected shared resource access, or systemcall interrupt. For example, the replay session has diverged before andis now entirely wrong. This is a replay error, The third possible resultis an expected stop condition. The replay can proceed and the next eventcan be de-queued from the log.

Thus, with the solution of the prior art patent application, all theinterrupts of multi-task applications are logged and replayed accuratelyin such a multi-processor environment.

However, logging too many events is costly and has a negative impact onperformances, especially with remote logging: event logging is costlyfrom an amount of storage point of view and for transferring theinformation from the recording machine to the replaying machine when itis remote; the impact on performances comes from the time to record andreplay and the time to transfer event information.

Within this model, it is also impossible to formally ensure that therewill be enough room in log fifos to store all the necessary events untilthe end of the scheduling period because one cannot predict how manysystem calls or exclusive accesses to shared resources will be performedbefore the release.

SUMMARY OF THE INVENTION

It is an object of the present invention to provide a reliable recordand replay function for non-deterministic events due to parallelenvironment, in particular, for multi-task applications.

It is a further object of the present invention to optimize theperformances of event record and replay by reducing the amount of eventinformation to be recorded on the recording machine, transferred to thereplaying machine if it is remote, and then used for replay on thereplaying machine.

The aforementioned aspects and other objectives and advantages can nowbe achieved as described herein. A method, system and computer programare disclosed for managing task events during the scheduling period of atask executing on one of the CPUs of a multi-processor computer. Onlyevents of specific portions of scheduling period are logged wherein afirst shared resource access has been granted for the task, this portionof scheduling period gathering all the non-deterministic events whichcannot be replayed by simple task re-execution. Other independentnon-deterministic event records are still logged as usual when theyoccur out of the portion of scheduling period for which a record hasbeen created. This limits the number of logged events during recordingsession of an application and the frequency of events to transmit fromthe production machine to the replay machine.

The principle of the invention is that the method distinguishes whichinterrupts are relevant or not (i.e., which ones need to be reproducedidentically to ensure deterministic replay and which ones can beignored). Only signals and schedule-out events occurring after theexclusive access resources are identically replayed. Also, severalnon-deterministic events are grouped into a unique event record. Thelast event of grouped records is a non-abortable event (NAE) that is anevent that modifies the state of the external world. Further, advantagescan be listed as follows: the implementation of the invention requireschanges in the operating system without any change requirement onhardware or record and replay machines. This technique is suitable forgeneric fault-tolerance systems. The new architecture described in thepresent invention improves the prior art architecture defined in theinternational patent application cited in the background art of thepresent document as follows: with this architecture, only interruptevents relevant to non-deterministic behavior are recorded. Thediscrimination of such events is based on type of resource involved(shared or not) and context of event (at start or within a schedulingperiod) used to arbitrate if we record the resource access details, orsimply the number of occurrences of resource access.

The disclosed approach is also capable of excluding data among what, forexample, constitutes a relevant scheduling period to be recorded andreplayed, such as, non-relevant data. It selects the event informationto record in order to have only one event per scheduling period insteadof many, which allows robustness of log resource checking. Compared tothe solution of the prior art described in the patent application citedin the background art, the solution of the invention simplifies thearchitecture by removing entirely the specific shm channel (QJShMPi inFIG. 9 of the international patent application WO 2006/077260). Asdescribed later in the document, the solution is based on distinguishingbetween ‘exclusive access period’ and non-exclusive access periodsincluded in the scheduling period of a task. Such exclusive accessperiod starts when the task is granted for the first time a sharedresource and ends when the last shared resource acquired during theperiod has been released by the activated task.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates the record process with the method of the prior art;

FIG. 2 illustrates the recording of event information during the taskexecution periods, wherein also illustrated are the successive ExclusiveAccess periods according to the preferred embodiment;

FIG. 3 illustrates the replay of event information during the taskexecution periods, wherein also illustrated are the successive ExclusiveAccess periods according to the preferred embodiment;

FIG. 4 illustrates a flowchart of the EAP event recording methodaccording to the preferred embodiment;

FIG. 5 illustrates a flowchart of the EAP event replaying methodaccording to the preferred embodiment; and

FIG. 6 illustrates the EAP event log during the recording session asillustrated in FIG. 2 according to the preferred embodiment.

DETAILED DESCRIPTION

FIG. 1 illustrates the record process with the method of the prior art.As illustrated in (FIG. 9) the international patent application WO2006/077260 of the prior art, during the recording session of tasksexecuting on a multiprocessor system, fifo queues are used to storeevent information. One fifo queue per each CPU (QJIProX, QJIProY) isused for logging each task schedule period information event and onefifo queue per shared resource (QJhMPi) is used for recording eachexclusive access to that shared resource during task execution. FIG. 1illustrates on one example the frequency of logging in the per CPU andper shared resource queues.

In this example of a recording session, an application of 2 processes Aand B run on a multi-processor machine, composed of processors CPUi andCPUj. A and B use shared resources SR1, SR2, SR3 (for example, 3distinct shared memory pages). On CPUi, a first scheduling period (100)starts when task A resumes and ends when the scheduler suspends. Duringthis first scheduling period, a first shared resource is accessed andinformation on this access is logged in the first shared resource queue(SR1), one second shared resource is also accessed and information onthis access is logged in the second shared resource queue (SR2). In thissame scheduling period, an event information is logged for a system callinterrupt (Syscall1) and for the Scheduler suspending task A in a CPUiqueue for the processor on which the corresponding system call isexecuted. During a second activation period (110) of task A, no sharedresource is accessed, only one second event information is logged for asystem call interrupt (Syscall2) in the same CPUi queue for theprocessor on which the corresponding system call is executed. During thethird period of activation (120), two system call events are logged intothe CPUi queue and one access to the first shared resource is logged inthe SR1 queue. Syscall4 is a blocking system call making a Schedulersuspends and SR1 released. During this time, task B is scheduled withfour scheduling periods (130, 140, 150, and 160) on a second processor.During the first activation period (130) for task B, the information onan access to a third shared resource is logged in a corresponding SR3fifo queue. The first activation period for task B ends when the task issuspended because it cannot access the first shared resource which isreserved by task A during its first period of activation. Task B resumeswhen the first shared resource is released by suspension of task A. Theinformation of the event of the first shared resource is reserved toprocess B is logged in the SR1 first shared resource queue. During theactivation period of process B, similarly to what is done during theactivation periods of process A, the event information are logged in theCPUj queue for the second processor on which process B is executed.

To improve the architecture of FIG. 1 for recording of applicationexecuting on a multiprocessor, it is noted that, in general, not all theinterrupts need to be logged and replayed, but only the ones related torelevant non-deterministic events at the application global state level.When applying this principle to the solution of the prior art, forexample, the scheduler interrupts, which define scheduling periods whereno shared resource was accessed, do not need to be recorded and replayedthe same, as they do not have any impact on the non-determinism of theapplication. Such events can be replayed by simple task codere-execution.

Now, still starting from the architecture for recording applicationsexecuting on a multiprocessor, considering shared memory accesses duringa scheduling period, only the first access needs to be logged. All thefollowing ones are implicit, even on different pages, if they occurwithin the same scheduling period. It is sufficient to record the numberof occurrences of a shared resource access, rather than all theoperation details.

The same applies for system calls, wherein we need to record only thenumber of occurrences rather than the full system call details.

A significant period for the process, an Exclusive Access Period (EAP),is herein defined. The EAP starts at the first successful access of ashared resource since the task resumed and ends when the task releasesthe last shared resource it has successfully exclusively accessed. TheEAP defines the period during which a task running on a CPU holdsexclusively one or more shared resources.

Shared resource exclusive access grants are released usually when thetimeslice expires, but this can also happen if a new non-deterministevent, not necessarily related to a shared resource, happens.

It can be advantageous, during the course of recording, to arbitrarilyend an EAP in order to ensure that the next event will be recordable. Onthe other hand, an EAP may accumulate many intermediate non-deterministevents as long as they are of fixed size and that space is available inthe log fifo. If the fifo becomes full, then the EAP is closed andcommitted in its current state and the event that could not be logged isused to open a new EAP. In summary, there is one recorded event perscheduling period. The event is recorded first in a local fifo beforebeing taken in charge by a logging task that is notified by changes inthe fifo.

FIG. 2 illustrates the recording of event information during the taskexecution periods, wherein also illustrated are the successive ExclusiveAccess periods (rectangles with hatchings 200, 210, 220 for process Aand 230, 240 for process B) according to the preferred embodiment. Thesame example is illustrated in FIG. 2 for a recording session as in FIG.1 of an application of 2 processes A and B running on a multi-processormachine, composed of processors CPUi and CPUj. A and B use sharedresources SR1, SR2, SR3 (for example, 3 distinct shared memory pages).On CPUi, the first EAP (200) during the first scheduling period forprocess A, is defined between the first access to SR1 and the schedulersuspends. Only one recorded event is generated, instead of 4 during thisfirst scheduling period of process A with the solution of prior artrecording each interrupt. During the second scheduling period of A, noshared resource is accessed and the system call Syscall2 doesn't carrynon-determinist information. Therefore, no event needs to be recordedfor this scheduling period. During the third scheduling period forprocess A, Syscall3 system call is logged in CPUi queue because it is anon-deterministic system call.

The event information logged is of variable size, as described in moredetails below.

There can be several reasons to end an EAP, that is to release holdshared resources and log an event record in the local fifo:

expiration of timeslice for the task, which is scheduled out;

blocking system call (for example, waiting for a response from aphysical device);

blocking access to a shared resource which is already held by aconcurrent task. This is detected by a failure to get exclusive accesslock;

no space available in local fifo for further event recording, therefore,inducing task blocking. This condition is checked at each sub-eventstart (entering a system call, hitting a new shared resource), thusallowing to record the already executed EAP;

beginning of an external I/O system call for which the indicated inputsize is larger than the available fifo space.

FIG. 3 shows a valid replaying session of a session already recorded asillustrated in FIG. 2. The replaying events are loaded from the fifoprior to resume the task, so a stop (STOP in FIG. 3) condition can beset. A stop condition matches at each interrupt. The stop conditiondefines the point in execution during replay at which the exclusiveaccess to shared resources must be released, in order to match exactlythe recorded EAP. The stop condition is read in advance, thus allowingan interrupt (either a performance counter overflow, or a breakpoint ata specific address in code) to be programmed, to force the execution tostop at the desired point. If the stop condition corresponds to theaccess to a system resource, or to a system call, then no furtherinstrumentation is required, as the task execution will already besuspended at the right location and the kernel handler just needs tocheck if the stop condition is reached or not.

FIG. 3 illustrates task execution at replay by reuse of the eventinformation stored in the processor queues CPUi and CPUj and transferredto the replay multiprocessor system. The execution for replay appearsdifferent from FIG. 2 in terms of task interleaving, but still behavesdeterministically as in FIG. 2, as the ordering of exclusive access tothe shared resources SR1 to SR3 is still maintained. However, thesequences of replay will be explained by the description of the newreplay algorithm as described later in the document in relation withdescription of FIG. 5.

The EAP illustration of FIG. 2 and FIG. 3 are obtained by using newevents, the EAP events and applying a new record and a replay algorithmusing the EAP event information.

With the new method for recording and replaying execution of processeson multiprocessor computers, only one fifo queue per processor is usedfor logging of event information. In the per processor queues, only theEAP events are logged, that is the information stored at the end of anEAP and some events out of an EAP which are non-deterministic:

the content of EAP event information logged at recording contains allthe necessary information;

a recording algorithm and a replay algorithm are applied, based on thelogging of EAP event information and reuse of this information forreplay. The EAP event variable size record logged in the CPU fifo at endof an EAP contains the following fields:

the number of system calls occurring during the EAP;

the number of shared resource access grants;

for each access grant, the shared resource unique id (SRID) and theshared resource sequence number (SRSN);

the number of user instructions with checksum and instruction pointer,the number of user instructions is relative to the last shared resourceaccess or system call, which of them occurs last;

the stop condition type (instruction count stop, system call stop, orshared resource access stop). No more shared resource queues are usedfor record/replay. It is noted that the shared resource channel “shm” ofthe prior art is suppressed, its role being now filled by the sharedresource sequence number (SRSN), which stores the global orderinginformation in the EAP record of the corresponding CPU channel.

During recording, the SRSN is incremented of 1 for each new EAPscheduling period start, as defined above. During replay, the constraintis to maintain the linear sequencing of SRSN whenever an event involvingshared resources occurs. This allows to synchronize the otherwisedecoupled CPU channels, exactly as it was done with the shm channel inthe prior art.

The SRSN starts from the first shared resource access, which marks thestart of an EAP. SRSN is associated in the EAP event information to theshared resource ID. This allows true parallelism as long as the sharedresources used by simultaneous tasks are uncoupled.

The events relating to other sources of non-determinism needs to berecorded in the CPU log channels:

IO system calls (pipes, socket, multiplexing . . . )

Storage access

External network events

Time, date.

All other events to be considered are (so far) associated to systemcalls, except some machine instructions used to read the clock (RDTSC).With the concept of EAP, all non-deterministic events can be categorizedin the following way:

a single non-deterministic event (NDE), which involves no sharedresources (for example, reading the time or computing a random number);

a non-deterministic event involving the access to a shared resource.This kind of event is part of an EAP as opposed to the previous one,which can or not be part of an EAP;

if an event involves output to an external system which cannot berollbacked, such as a shared storage or a network client, then thisevent is called a non-abortable event and the sequence of events between2 non-abortable events (not including the last one) is called anabortable sequence of events.

A consequence of this design is that event recording occurs notnecessarily at each system interrupt, but only at the end of a relevantscheduling period (for shared resource access related events) when allthe information about this scheduling period have been collected. Thereplay mechanism of prior art remains valid in its principle of checkingstop conditions.

There are no more shared resource channels, the “CPU” channels are nowsufficient to store alone all kinds of non-deterministic events, eachevent being complete and relevant. There is no impact on the processingof non-deterministic events related to resource virtualization (PIDidentifier under which a process is known by the Linux system and SHMIDwhich identifies the shared memory segment in Linux) (Linux is aregistered trademark of Linus Torvalds in the United States, othercountries, or both), or other resources.

The constraints brought by the requirement of providing fault-tolerancewithout interruption of service (in the case of record and replaymachines both active) are:

considering a task that is under control of the recording layer(interrupted in kernel mode), it must not be resumed until thenon-determinist event that induced the current state can be loggedlocally;

considering that an external communication constitutes a non-returnpoint in the execution flow, an external output from the applicationmust not be sent until all the locally recorded events up to the outputevent itself are committed to the standby machine on which the replaywill be performed.

A strong characteristic of the method according to the preferredembodiment is that all non-determinist events occur when in kernel mode,by the means of interrupts (system call or exceptions), where they canbe detected, recorded and forced (at replay) in a transparent manner forthe application. If it is not the default, the operating system isforced to behave this way (ex: shm access control).

Another interesting property of non-determinist events in kernel spaceis that their nature can be assessed before the event actually occurs.For example, it can be identified that that the next instruction is ashared memory access or is a system call with a given set of parameters.This allows checking if the next NDE is of fixed size (thereforeallowing the extension of EAP) or not.

As a consequence, it is better to let the event happen entirely thenattempt to record it, rather than having to reserve some space inadvance, when it is detected that the next instruction isnon-determinist.

FIG. 4 illustrates a flowchart of the EAP event recording methodaccording to the preferred embodiment. At initial state (400) when thetask is scheduled, it is not in an extended access period as all theshared resources are released. If the current number of shared resourceaccess grant is 0, and the task gets the exclusive access to a sharedresource (answer yes to test 410), then this is the start of an EAP. Anew record is started (420).

If an EAP is started for the task, a new shared resource exclusiveaccess is granted (answer yes to test 430), and if enough space (answeryes to test 450) for 1 sub-record (SRID, SRSN), then the number ofgrants is incremented, the sub-record is added to the CPU fifo (455),and the same EAP continues. If there is no space available (answer no totest 450), then the EAP is ended (460). Shared resources are releasedand the current EAP record is committed (470). The stop condition typeis, therefore, set to “shared resource access”.

If an EAP is started for the task and a system call is intercepted andif the syscall (based on the system call number) does not require aspecific treatment (answer yes to 440), then the number of interceptedsystem call is incremented and the same EAP continues.

If an EAP is started for the task and if a variable size event of adifferent type is about to start (i.e. a storage input system call, or anetwork input system call) (answer yes to test 480), then the EAP isfirst closed and the event is recorded (490). The stopping condition(system call start, shared resource access, or timeslice expiration) isrecorded. Multiple simultaneous recording and replay sessions areallowed. Each session has its own virtual namespace for system resourcelDs (PIDs, SHMIDs . . . ), allowing overlap without interference.

A replay session defines as many virtual CPUs as were recorded. Eventsare de-queued from each CPU log in the corresponding session virtual CPUdata structure. The initial starting event is, for instance, with Linuxoperating system, an execve system call which executes a process, with aper task sequence number set to 0 and a parent PID un-virtualized(forced to 1). The physical CPU not necessarily matches the virtual CPU.There is no control of inter-CPU migration of tasks during replay (orduring record). For that reason, all the CPU log queues are scanned toretrieve the next event corresponding to the virtual task. A per tasksequence number, having been set at recording to 0 at task creation(fork function in Linux to create a new process), then incremented ateach task recorded event, provides the necessary ordering information,in case of inter-CPU task migration during the log (2 or more CPU logsmatching the same virtual process identifier, PID in Linux).

At log event scan, if it matches the virtual PID of the task and thesequence number, then the event record is copied from the session'slogical CPU record into the task information control block, forinstance, the “cpulog_rec_t” data structure of the task descriptor“task_struct” in Linux. From that point, a new record from the virtualCPU log stream can be consumed.

FIG. 5 illustrates a flowchart of the replaying method according to thepreferred embodiment. The EAP event record loaded into the taskdescriptor allows determining the next stop condition (520) for thetask. This has to be evaluated before resuming the task. The stopcondition specifies how many system call interrupts, exclusive sharedresource access grants and number of user instructions since the latestof previous ones, which need to be performed by the task prior toprocess the next event.

If the number of system calls and share resource access grants remainingfor the EAP is 0 (answer yes to test 585), and if the stop condition isa user instruction count, then the UIC overflow/breakpoint proceduremust be setup (520) prior to resume the task. The task will be suspendedafter the right number of user instructions.

Case of shared resource access order inversion: if the current number ofshared resource access grants is 0 (in the task structure), then this isthe first access (500) and the task is suspended (510) until the pair(shared resource id, shared resource sequence number) matches thecurrent log record for the virtual CPU. When the match is reached, thecurrent log record is consumed:

copied into the task structure (520)

the current shared resource sequence number is incremented (530) and thetask is granted access (540).

If the current number of shared resource access grants is >1 (answer yesto test 550), then this number is decremented in the task descriptor(560) and the task is granted access (570). If the current number ofshared resource access grants is 1 (answer yes to test 580) and thecurrent number of system calls is >0 (in the task structure) (answer noto test 585), then the first number is zeroed (560) and the task isgranted access.

If the current number of shared resource access grants is 1 (answer yesto test 580) and the current number of system calls is 0 (answer yes totest 585), then the first number is zeroed and the task is set to stopafter the right number of user instructions (UIC) (590).

FIG. 6 illustrates the EAP event logged during the recording session asillustrated in FIG. 2, according to the preferred embodiment. During therecording session, the first EAP event logged (600) is EAP1 (200) fortask A on CPUi with one deterministic system call (Syscall1), twogranted accesses to shared memory resources (SR1 and SR2 in this order)and the scheduler suspends at time slice; this will be the stopcondition at replay. The second EAP event logged (610) is EAP2 (220) fortask A on CPUi, which includes no deterministic system call, one grantedshared resource SR1 and a stop condition with a blocking system call(Syscall4). A third EAP event logged (620) is EAP3 (230) for task B onCPUj which includes one deterministic system call (Syscall2), onegranted shared resource SR3 and a stop condition on tempting to access asecond shared resource SR1. A fourth EAP is logged (630), EAP4 (240) fortask B on CPUj, which includes no deterministic system call, one grantedshared resource SR1 and a stop condition for time slice,

With the comprehension of the method for replaying the EAP events onecan understand the replay sequence as illustrated in FIG. 3. Forinstance, the CPUi fifo content is read and each EAP event logged in theCPUi fifo for task A on CPUi is replayed. EAP1 as described in FIG. 6 isreplayed. The task descriptor containing the counters of the EAP eventdecremented each time a deterministic system call or a shared resourceaccess is granted and is saved at each time slice of the scheduler (300,310) or each time a stop condition (320) is replayed. When the schedulerof the replaying machine resumes task A, the updated counters of the EAPevent in the task descriptor are used to go on the EAP sequence (210)until a new time slice on the CPU of the replaying machine occurs or thestop condition of the replayed EAP event is reached.

It is noted that the usual non-deterministic events occurring during anactivation period of a task without any shared resource access grantedare stored independently of EAP events in the CPU fifo and replayed asusual. One example is the Syscall3 event which is logged as anindependent event and replayed after EAP1 and before EAP2 during thereplay session illustrated in FIG. 3.

This method preferably implemented as a record and a replay programincluding instrumentation of kernel, fits the need for generic faulttolerance systems allowing on the fly switch of one application to oneproduction machine to one replay machine. This does not prevent usingalso this solution for program debugging purpose on the same machine.

1. A method for managing task events during a scheduling period of atask executing on at least one CPU of a multi-processor computer, themethod comprising: detecting when a first access to a shared resource isgranted, setting a counter of number of granted access to a sharedresource to 1 and saving the shared resource id and 1 as the order inwhich this shared resource has been granted access during thisscheduling period; until one detection of an event preventing holding ofthe shared resource for which access has been already granted,incrementing a counter for each non-deterministic system call;incrementing the counter for number of granted access to a sharedresource each time a new access to a shared resource is granted andsaving the shared resource id and 1 as the order in which this sharedresource has been granted access during this scheduling period; upondetection of an event preventing holding of the shared resource forwhich access has been already granted, releasing hold shared resources;and logging in a fifo queue associated to the CPU on which the task isexecuting, a record comprising the number of deterministic system calls,the number of shared resource access grants, and, for each accessgranted, the shared resource id and the order of granted access, thenumber of instructions with checksum and instruction pointer in the codeof the task, relative to the last system call or granted access to ashared resource which of them occurs last and the type of eventpreventing holding of the shared resource for which access has beenalready granted.
 2. The method of claim 1 in which the detection of anevent preventing holding of the shared resource for which access hasbeen already granted, comprises detection of: end of scheduling periodfor the task upon expiration of timeslice; a blocking system call; ablocking access to a shared resource already hold by a concurrent task;and lack of place in fifo.
 3. The method of claim 1 further comprising,during the scheduling period of the task and if a first granted accessto a shared resource has not been detected, upon occurrence of anon-deterministic event, recording the non-deterministic eventinformation in the fifo queue associated to the CPU on which the task isexecuting.
 4. The method claim 1 further comprising making the fifoqueue content available to one multi-processor computer for replay. 5.The method of claim 4 in which the step of making the fifo queue contentavailable to one multi-processor computer for replay comprisestransmitting the fifo queue content to a multi-processor standbymachine.
 6. The method of claim 1 further comprising during schedulingperiod of a task replayed on one of the CPUs of a replayingmulti-processor computer: detecting when a first access to a sharedresource is granted for the task and suspending the task; reading in thefifo queue content the corresponding access granted to the sharedresource in the current record done during a scheduling period of a taskexecuting on one of the CPUs of a multi-processor computer, such recordnot being a non-deterministic event record; copying the record contentin the task descriptor; preparing a stop condition corresponding to thetype of event preventing holding of the shared resource for which accesshas been already granted read in the record; incrementing the order ofgranted access to the shared resource; resuming the task; grantingshared resource access; until the stop condition is reached, at eachgranted access to a shared resource or at each deterministic system calldecrementing the corresponding counters in the task descriptor; and upondetection of stop condition, forcing a stop to task stopping the taskexecution ending the corresponding part of scheduling period asrecorded.
 7. The method of claim 6 further comprising: detecting in thetask descriptor if the counter of granted access to shared resource andthe counter of system calls are null in the task descriptor; if the stopcondition is a not a timeslice stopping task execution after executingthe number of instructions relative to the last system call or grantedaccess to a shared resource, which of them occurred last read in thetask descriptor; and if the stop condition is a timeslice, schedulingperiod of the task ends with an UIC overflow set up in the step for whenpreparing stop condition.
 8. The method of claim 6 further comprising:detecting in the task descriptor if the counter of granted access toshared resource is greater than 1 the counter is decremented and theaccess to shared resource is granted.
 9. The method of any one of claim6 further comprising: detecting in the task descriptor if the counter ofgranted access to shared resource is 1 and the counter of system callis >0, then the counter of granted access to shared resource is zeroedand the access to shared resource is granted.
 10. The method of claim 6further comprising each time the task is suspended by a timeslice by thescheduler on CPUs of the replaying multi processor computer, the taskdescriptor is saved and when the scheduler resumes the task the replaycan go on until a new timeslice by the scheduler on CPUs of thereplaying multi-processor computer occurs or the stop condition of therecord is reached.
 11. A system for managing task events during ascheduling period of a task executing on at least one CPU of amulti-processor computer, the system comprising: a processor; a data buscoupled to the processor; and a computer-usable medium embodyingcomputer code, the computer-usable medium being coupled to the data bus,the computer program code comprising instructions executable by theprocessor and configured for: detecting when a first access to a sharedresource is granted, setting a counter of number of granted access to ashared resource to 1 and saving the shared resource id and 1 as theorder in which this shared resource has been granted access during thisscheduling period; until one detection of an event preventing holding ofthe shared resource for which access has been already granted,incrementing a counter for each non-deterministic system call;incrementing the counter for number of granted access to a sharedresource each time a new access to a shared resource is granted andsaving the shared resource id and 1 as the order in which this sharedresource has been granted access during this scheduling period; upondetection of an event preventing holding of the shared resource forwhich access has been already granted, releasing hold shared resources;and logging in a fifo queue associated to the CPU on which the task isexecuting, a record comprising the number of deterministic system calls,the number of shared resource access grants, and, for each accessgranted, the shared resource id and the order of granted access, thenumber of instructions with checksum and instruction pointer in the codeof the task, relative to the last system call or granted access to ashared resource which of them occurs last and the type of eventpreventing holding of the shared resource for which access has beenalready granted.
 12. The system of claim 11 in which the detection of anevent preventing holding of the shared resource for which access hasbeen already granted, comprises detection of: end of scheduling periodfor the task upon expiration of timeslice; a blocking system call; ablocking access to a shared resource already hold by a concurrent task;lack of place in fifo.
 13. The system of claim 11 further comprising,during the scheduling period of the task and if a first granted accessto a shared resource has not been detected, upon occurrence of anon-deterministic event, recording the non-deterministic eventinformation in the fifo queue associated to the CPU on which the task isexecuting.
 14. The system of claim 11 further comprising making the fifoqueue content available to one multi-processor computer for replay. 15.The system of claim 14 in which the step of making the fifo queuecontent available to one multi-processor computer for replay comprisestransmitting the fifo queue content to a multi-processor standbymachine.
 16. The system of claim 11 further comprising during schedulingperiod of a task replayed on one of the CPUs of a replayingmulti-processor computer: detecting when a first access to a sharedresource is granted for the task and suspending the task; reading in thefifo queue content the corresponding access granted to the sharedresource in the current record done during a scheduling period of a taskexecuting on one of the CPUs of a multi-processor computer, such recordnot being a non-deterministic event record; copying the record contentin the task descriptor; preparing a stop condition corresponding to thetype of event preventing holding of the shared resource for which accesshas been already granted read in the record; incrementing the order ofgranted access to the shared resource; resuming the task; grantingshared resource access; until the stop condition is reached, at eachgranted access to a shared resource or at each deterministic system calldecrementing the corresponding counters in the task descriptor; and upondetection of stop condition, forcing a stop to task stopping the taskexecution ending the corresponding part of scheduling period asrecorded.
 17. A computer-usable medium for managing task events during ascheduling period of a task executing on at least one CPU of amulti-processor computer, the computer-usable medium embodying computerprogram code, the computer program code comprising computer executableinstructions configured for: detecting when a first access to a sharedresource is granted, setting a counter of number of granted access to ashared resource to 1 and saving the shared resource id and 1 as theorder in which this shared resource has been granted access during thisscheduling period; until one detection of an event preventing holding ofthe shared resource for which access has been already granted,incrementing a counter for each non-deterministic system call;incrementing the counter for number of granted access to a sharedresource each time a new access to a shared resource is granted andsaving the shared resource id and 1 as the order in which this sharedresource has been granted access during this scheduling period; upondetection of an event preventing holding of the shared resource forwhich access has been already granted, releasing hold shared resources;and logging in a fifo queue associated to the CPU on which the task isexecuting, a record comprising the number of deterministic system calls,the number of shared resource access grants, and, for each accessgranted, the shared resource id and the order of granted access, thenumber of instructions with checksum and instruction pointer in the codeof the task, relative to the last system call or granted access to ashared resource which of them occurs last and the type of eventpreventing holding of the shared resource for which access has beenalready granted.
 18. The computer-usable medium of claim 17 in which thedetection of an event preventing holding of the shared resource forwhich access has been already granted, comprises detection of: end ofscheduling period for the task upon expiration of timeslice; a blockingsystem call; a blocking access to a shared resource already hold by aconcurrent task; and lack of place in fifo.
 19. The computer-usablemedium of claim 17 further comprising, during the scheduling period ofthe task and if a first granted access to a shared resource has not beendetected, upon occurrence of a non-deterministic event, recording thenon-deterministic event information in the fifo queue associated to theCPU on which the task is executing.
 20. The computer-usable medium claim17 further comprising making the fifo queue content available to onemulti-processor computer for replay.